BiDi screen : depth and lighting aware interaction and display

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 75-79).In this thesis, I describe a new type of interactive display that supports both on-screen multi-touch interactions and off-screen hover-based gestures. This BiDirectional (BiDi) screen, capable of both image capture and display, is inspired by emerging LCDs that use embedded optical sensors to detect multiple points of direct contact. The key contribution of this thesis is to exploit the spatial light modulation capability of LCDs to allow dynamic mask-based scene capture without interfering with display functionality. A large-format image sensor is placed slightly behind the liquid crystal layer. By alternatly switching the liquid crystal between a display mode showing traditional graphics and a capture mode in which the backlight is disabled and a pinhole array or an equivalent tiled-broadband code is displayed, the BiDi Screen can recover multi-view orthographic imagery while functioning as a 2D display. The recovered imagery is used to passively estimate the depth of scene points from focus. I discuss the design and construction of a prototype to demonstrate these capabilities in two motivating applications: a hybrid touch plus gesture interaction and a light-gun mode for interacting with external light-emitting widgets. The working prototype simulates the large format light sensor with a camera and diffuser, supporting interaction up to 50 cm in front of a modified 20.1 inch LCD.by Matthew W. Hirsch.S.M

A compressive light field projection system

For about a century, researchers and experimentalists have strived to bring glasses-free 3D experiences to the big screen. Much progress has been made and light field projection systems are now commercially available. Unfortunately, available display systems usually employ dozens of devices making such setups costly, energy inefficient, and bulky. We present a compressive approach to light field synthesis with projection devices. For this purpose, we propose a novel, passive screen design that is inspired by angle-expanding Keplerian telescopes. Combined with high-speed light field projection and nonnegative light field factorization, we demonstrate that compressive light field projection is possible with a single device. We build a prototype light field projector and angle-expanding screen from scratch, evaluate the system in simulation, present a variety of results, and demonstrate that the projector can alternatively achieve super-resolved and high dynamic range 2D image display when used with a conventional screen.MIT Media Lab ConsortiumNatural Sciences and Engineering Research Council of Canada (NSERC Postdoctoral Fellowship)National Science Foundation (U.S.) (Grant NSF grant 0831281

Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting

We introduce tensor displays: a family of compressive light field displays comprising all architectures employing a stack of time-multiplexed, light-attenuating layers illuminated by uniform or directional backlighting (i.e., any low-resolution light field emitter). We show that the light field emitted by an N-layer, M-frame tensor display can be represented by an Nth-order, rank-M tensor. Using this representation we introduce a unified optimization framework, based on nonnegative tensor factorization (NTF), encompassing all tensor display architectures. This framework is the first to allow joint multilayer, multiframe light field decompositions, significantly reducing artifacts observed with prior multilayer-only and multiframe-only decompositions; it is also the first optimization method for designs combining multiple layers with directional backlighting. We verify the benefits and limitations of tensor displays by constructing a prototype using modified LCD panels and a custom integral imaging backlight. Our efficient, GPU-based NTF implementation enables interactive applications. Through simulations and experiments we show that tensor displays reveal practical architectures with greater depths of field, wider fields of view, and thinner form factors, compared to prior automultiscopic displays.United States. Defense Advanced Research Projects Agency (DARPA SCENICC program)National Science Foundation (U.S.) (NSF Grant IIS-1116452)United States. Defense Advanced Research Projects Agency (DARPA MOSAIC program)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award)Alfred P. Sloan Foundation (Fellowship

Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization

We optimize automultiscopic displays built by stacking a pair of modified LCD panels. To date, such dual-stacked LCDs have used heuristic parallax barriers for view-dependent imagery: the front LCD shows a fixed array of slits or pinholes, independent of the multi-view content. While prior works adapt the spacing between slits or pinholes, depending on viewer position, we show both layers can also be adapted to the multi-view content, increasing brightness and refresh rate. Unlike conventional barriers, both masks are allowed to exhibit non-binary opacities. It is shown that any 4D light field emitted by a dual-stacked LCD is the tensor product of two 2D masks. Thus, any pair of 1D masks only achieves a rank-1 approximation of a 2D light field. Temporal multiplexing of masks is shown to achieve higher-rank approximations. Non-negative matrix factorization (NMF) minimizes the weighted Euclidean distance between a target light field and that emitted by the display. Simulations and experiments characterize the resulting content-adaptive parallax barriers for low-rank light field approximation.National Science Foundation (U.S.) (grant CCF-0729126)National Research Foundation of Korea (grant 2009-352-D00232

BiDi screen: a thin, depth-sensing LCD for 3D interaction using light fields

We transform an LCD into a display that supports both 2D multi-touch and unencumbered 3D gestures. Our BiDirectional (BiDi) screen, capable of both image capture and display, is inspired by emerging LCDs that use embedded optical sensors to detect multiple points of contact. Our key contribution is to exploit the spatial light modulation capability of LCDs to allow lensless imaging without interfering with display functionality. We switch between a display mode showing traditional graphics and a capture mode in which the backlight is disabled and the LCD displays a pinhole array or an equivalent tiled-broadband code. A large-format image sensor is placed slightly behind the liquid crystal layer. Together, the image sensor and LCD form a mask-based light field camera, capturing an array of images equivalent to that produced by a camera array spanning the display surface. The recovered multi-view orthographic imagery is used to passively estimate the depth of scene points. Two motivating applications are described: a hybrid touch plus gesture interaction and a light-gun mode for interacting with external light-emitting widgets. We show a working prototype that simulates the image sensor with a camera and diffuser, allowing interaction up to 50 cm in front of a modified 20.1 inch LCD.National Science Foundation (U.S.) (Grant CCF-0729126)Alfred P. Sloan Foundatio

Polarization fields: dynamic light field display using multi-layer LCDs

We introduce polarization field displays as an optically-efficient design for dynamic light field display using multi-layered LCDs. Such displays consist of a stacked set of liquid crystal panels with a single pair of crossed linear polarizers. Each layer is modeled as a spatially-controllable polarization rotator, as opposed to a conventional spatial light modulator that directly attenuates light. Color display is achieved using field sequential color illumination with monochromatic LCDs, mitigating severe attenuation and moiré occurring with layered color filter arrays. We demonstrate such displays can be controlled, at interactive refresh rates, by adopting the SART algorithm to tomographically solve for the optimal spatially-varying polarization state rotations applied by each layer. We validate our design by constructing a prototype using modified off-the-shelf panels. We demonstrate interactive display using a GPU-based SART implementation supporting both polarization-based and attenuation-based architectures. Experiments characterize the accuracy of our image formation model, verifying polarization field displays achieve increased brightness, higher resolution, and extended depth of field, as compared to existing automultiscopic display methods for dual-layer and multi-layer LCDs.National Science Foundation (U.S.) (Grant IIS-1116452)United States. Defense Advanced Research Projects Agency (Grant HR0011-10-C-0073)Alfred P. Sloan Foundation (Research Fellowship)United States. Defense Advanced Research Projects Agency (Young Faculty Award

Focus 3D: Compressive Accommodation Display

We present a glasses-free 3D display design with the potential to provide viewers with nearly correct accommodative depth cues, as well as motion parallax and binocular cues. Building on multilayer attenuator and directional backlight architectures, the proposed design achieves the high angular resolution needed for accommodation by placing spatial light modulators about a large lens: one conjugate to the viewer's eye, and one or more near the plane of the lens. Nonnegative tensor factorization is used to compress a high angular resolution light field into a set of masks that can be displayed on a pair of commodity LCD panels. By constraining the tensor factorization to preserve only those light rays seen by the viewer, we effectively steer narrow high-resolution viewing cones into the user's eyes, allowing binocular disparity, motion parallax, and the potential for nearly correct accommodation over a wide field of view. We verify the design experimentally by focusing a camera at different depths about a prototype display, establish formal upper bounds on the design's accommodation range and diffraction-limited performance, and discuss practical limitations that must be overcome to allow the device to be used with human observers

A switchable light field camera architecture with Angle Sensitive Pixels and dictionary-based sparse coding

We propose a flexible light field camera architecture that is at the convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, Angle Sensitive Pixels and advanced reconstruction algorithms, we show that-contrary to light field cameras today-our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.National Science Foundation (U.S.) (NSF Grant IIS-1218411)National Science Foundation (U.S.) (NSF Grant IIS-1116452)MIT Media Lab ConsortiumNational Science Foundation (U.S.) (NSF Graduate Research Fellowship)Natural Sciences and Engineering Research Council of Canada (NSERC Postdoctoral Fellowship)Alfred P. Sloan Foundation (Research Fellowship)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award

Computational visual reality

Thesis: Ph. D., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2014.Cataloged from PDF version of thesis.Includes bibliographical references (pages 231-245).It is not so far-fetched to envision a future student working through a difficult physics problem by using their hands to manipulate a 3D visualization that floats above the desk. A doctor preparing for heart surgery will rehearse on a photo-real replica of his patient's organ. A visitor to the British Museum in London will sketch a golden Pharaoh's headdress, illuminated by a ray of sunlight pouring in the window, never aware that the physical artifact is still in Egypt. Though such scenarios may seem cut from the pages of science fiction, this thesis illuminates a path to making them possible. To create more realistic and interactive visual information, displays must show high quality 3D images that respond to environmental lighting conditions and user input. The availability of displays capable of addressing the full range of visual experience will improve our ability to interact with computation, the world, and one another. Two of the many problems that have impeded previous efforts to design high-dimensional displays are the need to: 1. process large amounts of information in realtime; and 2. fabricate hardware capable of conveying that information. Light field capture and display is enormously data-intensive, but by applying compressive techniques that take advantage of multiple data redundancies in light transport, it is possible to overcome these challenges and make use of hardware available in the near-term. This thesis proposes display and capture frameworks that use non-negative tensor factorization and dictionary-based sparse reconstruction, respectively, in conjunction with the co-design of algorithms, optics, and electronics to allow compressive, simultaneous, light field display and capture.by Matthew Waggener Hirsch.Ph. D

8D display: A Relightable Glasses-Free 3D Display

magine a display that behaves like a window. Glancing through it, viewers perceive a virtual 3D scene with correct parallax, without the need to wear glasses or track the user. Light that passes through the display correctly illuminates the virtual scene. While researchers have considered such displays, or prototyped subsets of these capabilities, we contribute a new, interactive, relightable, glasses-free 3D display. By simultaneously capturing a 4D light field, and displaying a 4D light field, we are able to realistically modulate the incident light on rendered content. We present our optical design, and GPU pipeline. Beyond mimicking the physical appearance of objects under natural lighting, an 8D display can create arbitrary directional illumination patterns and record their interaction with physical objects. Our hardware points the way towards novel 3D interfaces, in which users interact with digital content using light widgets, physical objects, and gesture. © 2012 Authors